Skip to content

Supports Asynchronous Runs in Interactive Beam #36853

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

Supports Asynchronous Runs in Interactive Beam #36853

wants to merge 8 commits into from
+1,521 −19

Conversation

@ian-Liaozy
Copy link
Contributor

@ian-Liaozy ian-Liaozy commented Nov 19, 2025

Fixes #33103

Implement the feature request of adding asynchronous computation to Apache Beam's Interactive Beam API. This means allowing long-running tasks to execute in the background without blocking the user interface.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@ian-Liaozy ian-Liaozy marked this pull request as ready for review November 19, 2025 18:44
@ian-Liaozy
Copy link
Contributor Author

ian-Liaozy commented Nov 19, 2025

Run Python PreCommit

@ian-Liaozy
Copy link
Contributor Author

ian-Liaozy commented Nov 19, 2025

Run Python PreCommit

this doesn't seem to work

@codecov
Copy link

codecov bot commented Nov 19, 2025

Codecov Report

❌ Patch coverage is 13.28904% with 261 lines in your changes missing coverage. Please review.
✅ Project coverage is 40.63%. Comparing base (c8d7ca0) to head (8deb0f4).
⚠️ Report is 29 commits behind head on master.

Files with missing lines Patch % Lines
...ache_beam/runners/interactive/recording_manager.py 12.01% 227 Missing ⚠️
...pache_beam/runners/interactive/interactive_beam.py 9.09% 30 Missing ⚠️
...eam/runners/interactive/interactive_environment.py 60.00% 4 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##             master   #36853      +/-   ##
============================================
+ Coverage     40.33%   40.63%   +0.29%     
  Complexity     3456     3456              
============================================
  Files          1225     1225              
  Lines        187817   188122     +305     
  Branches       3586     3586              
============================================
+ Hits          75762    76439     +677     
+ Misses       108665   108293     -372     
  Partials       3390     3390
Flag Coverage Δ
python 41.28% <13.28%> (+0.70%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 19, 2025

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@ian-Liaozy ian-Liaozy force-pushed the users/ianliao/ib-async-run branch from e3233d7 to dd386ea Compare November 19, 2025 23:58
@ian-Liaozy ian-Liaozy force-pushed the users/ianliao/ib-async-run branch from 3bea653 to 576acab Compare November 20, 2025 02:34
@ian-Liaozy
Copy link
Contributor Author

ian-Liaozy commented Nov 20, 2025

assign set of reviewers

@github-actions
Copy link
Contributor

github-actions bot commented Nov 20, 2025

Assigning reviewers:

R: @jrmccluskey for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@jrmccluskey
Copy link
Contributor

jrmccluskey commented Nov 20, 2025

Run Python PreCommit

this doesn't seem to work

Trigger phrases don't work since we're no longer picking up jobs to run on Jenkins, there are JSON trigger files for workflows in the .github directory that you can modify to force runs of suites now

jrmccluskey
jrmccluskey reviewed Nov 20, 2025
View reviewed changes
Copy link
Contributor

@jrmccluskey jrmccluskey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code is broadly LGTM, have you run it in a notebook environment? Would love to see what the user experience is like before merging

@ian-Liaozy
Copy link
Contributor Author

ian-Liaozy commented Nov 20, 2025

Code is broadly LGTM, have you run it in a notebook environment? Would love to see what the user experience is like before merging

Will send it to you

@ian-Liaozy ian-Liaozy force-pushed the users/ianliao/ib-async-run branch from ffeebb7 to e7eb7c9 Compare November 21, 2025 00:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jrmccluskey jrmccluskey Awaiting requested review from jrmccluskey

Assignees

No one assigned

Labels

Next Action: Reviewers python runners

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature Request]: Interactive Beam supports asynchronous computations.

2 participants

@ian-Liaozy @jrmccluskey